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Abstract 

The purpose of the paper is to give new key agreement protocols (a multi-party 
extension of the protocol due to Anshel-Anshel-Goldfeld and a generalization of the 
Diffie-Hellman protocol from abelian to solvable groups) and a new homomorphic 
public-key cryptosystem. They rely on difficulty of the conjugacy and membership 
problems for subgroups of a given group. To support these and other known cryp- 
tographic schemes we present a general technique to produce a family of instances 
being matrix groups (over finite commutative rings) which play a role for these 
schemes similar to the groups Z* in the existing cryptographic constructions like 
RSA or discrete logarithm. 

*Partially supported by RFFI, grants, 03-01-00349, NSH-2251. 2003.1. The paper was done during the 
stay of the author at the Mathematical Institute of the University of Rennes. 
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Introduction 



One of the oldest cryptographical problems consists in constructing of a key agreement 
protocol. Roughly speaking it is a multi-party algorithm, defined by a sequence of steps, 
specifying the actions of two or more parties in order a shared secret becomes available 
to two or more parties. Probably the first such procedure based on abelian groups is 
due to Diffie-Hellman one (see [7j). In fact, it concerns automorphisms of abelian (even 
cyclic) groups induced by taking to a power. Some generalizations of this protocol to non- 
abelian groups (in particular, the matrix groups over some rings) were suggested in [2*T] 
where security was based on an analog of the discrete logarithm problems in groups of 
inner automorphisms. Certain variations of the Diffie-Hellman systems over the braid 
groups were described in there several trapdoor one-way functions connected with 
the conjugacy and the taking root problems in the braid groups were proposed. Recently, 
a general scheme for constructing key agreement protocols based on algebraic structures 
was proposed in [T] . In principle, it enables us to construct such protocols for non-abelian 
groups and their automorphisms induced by conjugations. In this paper we generalize 
to the non-abelian case the Diffie-Hellman protocol, construct multi party procedure for 
the protocol £Q , and analyze the security of both protocols realized in matrix groups over 
rings. 

The question on finding probabilistic public-key cryptosystems in which the decryp- 
tion function has a homomorphic property goes back to [22] (see also jOJ). In such a 
cryptosystem the spaces of messages and of ciphertexts are algebraic structures G and 
H and the decryption function D : G — > H is a homomorphism. A number of such 
cryptosystems is known for abelian groups, e.g. the quadratic residue cryptosystem jTj 
and its generalization for highest residues [2U| ( see a ^ so an overview in |10j). In most of 
them the security is based on the intractability of theoretical number problems close to 
the integer factoring. Recently, several homomorphic cryptosystems were constructed for 
infinite (but finitely presented) groups, see [TUl HI] and references there. In this paper we 
construct one more homomorphic cryptosystem with G being a free group the trapdoor 
of which uses a secret permutation of the generators of G. 

The third problem considered in this paper is how to produce instances for cryptosys- 
tems based on computations with matrix groups over rings. In contrast to numerous 
theoretical cryptosystems where there is a lot of efficient algorithms to generate integers 
with given properties (e.g., the pairs of two distinct large primes of the same bit size 
used in the quadratic residue cryptosystem), it is not clear a priory how to find efficiently 
matrix groups in which some problems (like membership or conjugacy) arising in cryptog- 
raphy are computationally difficult. We propose a general scheme for solving this problem 
and give a specialization of this scheme for matrix groups over finite commutative rings. 

In Section ^ we study key agreement protocols between two parties (named usually 
Alice and Bob). The security of the Diffie-Hellman protocol relies on the difficulty of the 
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following transporter problem: having an action G x V — > V of a group G on a set V for 
given it, v 6 V to find g & G (provided that it does exist) such that (g, u) \— > v. In case of 
V being a cyclic group of order n and G being a group acting on V by taking a power one 
arrives to the discrete logarithm problem (usually, n is taken to be prime). The security 
of the key agreement protocol of pQ (see also Subsection 11.1)) relies on the difficulty of 
the conjugacy problem with respect to a subgroup of G. In Subsection 11.11 we extend 
the construction of [1J to multi- party key agreement protocol. Then in Subsection II. 21 we 
design another generalization of the Diffie-Hellman protocol to actions of groups G which 
satisfy a certain identity. Clearly, any abelian group satisfies the identity a6a _1 6 _1 = 1 
and more generally, any solvable group with a fixed length of its derived series satisfies 
an appropriate commutator identity. The security of our protocol again relies on the 
difficulty of the transporter problem for a suitable action of G. 

In Section[2]we consider homomorphic public key cryptosystems (sec e.g. [10J) in which 
the decrypting function (known to Alice) is a group homomorphism / : G —>■ H where 
the groups H, G play the roles of the spaces of plain and ciphertext messages respectively. 
Usually, the security of a homomorphic cryptosystem relies on the difficulty of the problem 
of the membership to a normal subgroup of G (here, the kernel of /) . Also in Section |2] 
we describe a homomorphic cryptosystem in which as G a free group is taken. This 
cryptosystem modifies one from ^1] where as G a subgroup of the modular group SL^Z) 
was considered. The security of this cryptosystem relies on the difficulty of a certain word 
problem. A private key of Alice is an appropriate permutation of the generators of the 
free group G, this differs our cryptosystem from the one produced in [5]. 

The crucial role in the classical cryptographic constructions (like RSA, discrete loga- 
rithm or quadratic residue [Z]) plays the natural action of the group Aut(Z*) on the group 
Z*. So, varying n one gets a mass pool of instances for cryptographic primitives. This 
action is a special case of the natural action of the group Autn(V) (viewed as a matrix 
group) on the free module V over the ring R. In this paper we propose a construction 
of a pool of matrix groups instances for cryptographic primitives ( Subsection 13. 2|) . The 
security of these instances relies on the difficulty of certain problems on matrix groups 
(e.g. the membership to a subgroup or the conjugacy with respect to a subgroup). For 
the complexity of such problems few results were established in case of matrix groups over 
fields |3J ; for matrix groups over arbitrary rings much less is known. 

The common way in cryptography of producing a trapdoor and a cryptosystem, is to 
generate a private key departing from a pair of primes p, q, while their product n = pq 
plays the role of a public key. In our scheme (see Subsection 13. 1|) as a private key we take 
a rooted tree whose leaves being furnished with specially chosen (non-abelian, in general) 
groups. We assume that Alice has in possession such representations of these groups 
which allow her to solve efficiently a problem lying in the background of a cryptosystem 
(like membership or conjugacy). Internal vertices of the tree are endowed with certain 
operations on groups which allow one to assign recursively a group to each vertex of the 
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tree starting with its leaves. At the end of the recursion a group is assigned to the root, 
and this group plays the role of a public key. This scheme is also modified to produce a 
homomorphism of matrix groups as a public key. In Subsection 13.21 we give a realization 
of this general scheme in finite matrix groups. 

The similarity of the common constructions in cryptography based on commutative 
groups (say. Z* ) with our construction (relying on finite matrix groups) allows us to call 
the latter type of constructions the non- commutative cryptography. 

1 Group theoretical key agreement protocol 

1.1 A multi-party protocol. The following group theoretical variant of key agreement 
two-party protocol was proposed in [T|. Let G be a group, and to two parties A and B 
are assigned their subgroups 

G A = (a 1 ,...,a m ), G B =(&!,... ,b n ). (1) 

The group G and the elements a^, bj are publicly known. The parties A and B choose 
secret elements a £ Ga and b £ Gb and transmit to each other the collections 

X B = {a~ V)?=i, Xa = {b-\b}Zi 

respectively. Since A (resp. B) has a representation of the element a (resp. b) via 
generators a\, . . . , a m (resp. bi, . . . , b n ), then A (resp. B) can compute a representation 
of the element b~ x ab (resp. a~ l ba) via elements of the set Xa (resp. Xg). Thus A and B 
have a common key 

a^ib^ab) = [a,b] = (a' l ba)- l b. 

An obvious necessary condition for this protocol to be secure is that the set of all such 
commutators with a £ Ga, and b £ Gb would contain at least two elements. 

Let us describe a generalization of the group theoretical key agreement protocol for s 
parties with s > 2 and a single public communicating channel. Without loss of generality 
we assume that s = 2* for some t > 1, for otherwise in the recursive construction below 
we divide the parties into two unequal subsets which leads just to slight changing the 
notation. As in the case s = 2 the groups G\,...,G S C G of the parties are given 
publically by their sets of generators. At the initial step the zth party chooses a secret key 
Oj £ Gi, i — 1, . . . , s. Let Si and S 2 be disjoint s/2-subsets of the set {1, . . . , s}. Then 
given u = 1,2 the parties from S u recursively construct the common key K u £ G, such 
that for all % £ S u there exist integers £ { — 1,+1} and 1 < m ; < s/2, and certain 
elements B^i, . . . , Bi >rrH £ ({a? : j £ S u ^}) with S U} i = S u \ {i}, for which we have 
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By recursion we can assume that the ith party knows the elements B~^aB { ■ for all j and 
for all chosen generators a of the group Gi (and thereby, it knows B~ja i B i j)^ but does not 
necessary know B^y At this point the party i G S u sends the elements B^jaB i - for all 
the chosen generators a of the group Gi to a certain party from the set S u / with u' = 3 — u 
and asks for the elements K~}B^aB i ^K u ,. Then for u = 1 the ith party computes the 
element 

[if 1; K 2 ] = K^\K^K X K 2 ) = K^\K 2 \Br 1 m a £ r H B hmi )K 2 ) • ■ ■ {K^{B^B^)K^). 

Similarly, for u = 2 the ith party computes the element [if 1,^2] = (Jf{~ Jf 2 ifi) -1 if2- 
Thus this element can be chosen as the common key for all parties. It is easy to see that 
the ith party computes the common key in 0(s|a;|) operations in the group G, where |aj| 
denotes the length of the word a, in the chosen generators of the group Gi. 

1.2 A new protocol. In this subsection we define a new group-theoretical two party 
key agreement protocol that can be viewed as a non-commutative generalization of the 
Diffie-Hellman protocol (see [7j). 

Let G be a group acting on a set X so that given (x,g) G X x G the image x 9 of 
x with respect to g can be efficiently computed. Two parties A and B going to choose 
a secret common key from X, fix publically subgroups Ga,Gb of the group G and two 
words 

11/ / \ Ol,l 61 1 11, mi y t 7" / \ &2.1 a 2,l fe rno 

Wa{ua,ub)=u a 'u b ' ■■■u a l , Wb{Ua,U B ) = u B u A ' ■•■u B 
of the free group F 2 with two free generators ua,ub such that 
(Wl) mi, m 2 G N, Ojj, fojj G Z for all i, j, and ai j7ni 7^ 0, 6 2jm2 7^ 0, 
(W2) M^i^A, 9b) = W B (gA,9B) for all (jg A ,g B ) G G A x G B . 

The protocol begins with the choice of a publically known element xq E X and the secret 
elements gA G G A by the party A and #b G G# by the party B. Then during the 
communications the party A performs the following: 

- Set Ka = xq. 



- For i — 1, . . . , mi — 1 send if^ A '* and receive Ka '■= K 9 A A ' lgB 



9A 1,m l 



- Set A" A := if A 

The communications of the party B are defined similarly. Thus at the end of the com- 
munication process due to condition (W2) the parties A and B have the common key 

If _ W A {9A,9b) _ Wb(9A,9b) _ iy- 
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For X = Z* with p being a prime, G = Ga = Gb being the group Z*_ x = Aut(Z*) and 
W^4(ma,mb) = ubUa, Wb(ua,v>b) = uaUb we come to the Diffie-Hellman protocol. 

This scheme can be easily realized for a solvable group G with bounded length n of 
the derived series of G. For example, one can take Ga = Gb = G and choose the words 
Wa = Wa,u and Wb = Wb,u by induction on n as follows. If n — 1, then the group G is 
abelian and so conditions (Wl) and (W2) are satisfied for 

W A ,i(u A , u B ) = u B u A , W B ,i{u A , u B ) = u A u B . 

For n > 2 the commutator [g, h] = g~ 1 h~ 1 gh with arbitrary g,h G G belongs to the 
derived subgroup G' = [G, G] of G (the derived length of G' equals n — 1). Assume 
by induction that conditions (Wl) and (W2) are satisfied for the words Wa,u-i and 
Wb,u-\- Then a straightforward checking shows that these conditions are also satisfied, 
for example, for the words 

W A , n = W A ,n-l([u B ,U A ], [l^Slifl 1 ]), W B ,n = W B ,n-l ( [u B , «a] , l"^.^ 1 ])- 

This follows from the fact that the length (the number of letters) of the word W A , n ( as 
well as Wb,h) equals 2 • 4 n_1 which one can verify by induction on n > 1. More generally, 
to define and Ws, n one can choose arbitrary words Wi, W2, W3, W4 G Wx where 

X = {ua,ub} and Wx is the set of all words in the alphabet X ± , and use [W\, W%] and 
[W^jW^] instead of [u A , u B ) and [Mij 1 ,^^ 1 ] respectively. Certainly, to provide condition 
(1) one should guarantee that the words Wa,ti-i{ua, ub) (resp. Ws^-iiuA, ub)) and W 2 
(res. W4) must be terminated to ua (resp. u B )- To avoid triviality we also should take 
Wi, . . . , W4 so that Wa,u and would be nonidentity elements in the underlying free 
group. 

Clearly, any realization of the above protocol is based on identities of the group G. 
In addition to commutator identities for solvable groups (see above) one can also use the 
identity x m = 1 (that holds in any finite group the order of which is a divisor of m, and in 
the Burnside groups). In this case we can choose as the words W A and Wb the prefix and 
the inverse of the suffix of the word (uaUb) 171 , respectively, so that the prefix is terminated 
to ua- In fact, as it was proved by B.Neumann any variety of groups can be given by a 
collection of identities such that the first of them is of the form x m = 1 with m being 
a nonnegative integer, whereas the other ones are the elements of the commutant of the 
underlying free group (see [To]). 

We complete the subsection by making two remarks on the above protocol. First, 
the set X must be of superpolynomial size, for otherwise the key agreement scheme 
can be broken in polynomial time by the known permutation group theory technique 
(see [IS]). Second, the words W A and Wb must be chosen so that the number of elements 
W A {gAi 9b) — Ws(gA, Qb) with gA,9B G G would contain at least two elements. 
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1.3 On the security of the protocols. In the above protocols we assume that all 
groups are given explicitly, e.g. by sets of generators, so that the group operations can be 
performed efficiently Then the security of the first protocol is based on the intractability 
of the following problem (see [23] )■ 

Subgroup Conjugation Search Problem (SCSP). Given a group G, subgroups 
Hi,H 2 of G, and two elements f,g £ Hi, find an element h £ H 2 such that f = h^gh, 
provided that at least one such h exists. 

As usually in the cryptography, an efficient algorithm solving SCSP would break the 
protocol (but to break the protocol it is not necessary to solve SCSP). Such an algorithm 
does exist for G = GL(n, ¥ q ) where n is a natural number, W q is a finite field of the order q, 
and the subalgebra A(H 2 ) of the full matrix algebra Mat n (F g ) generated by the group H 2 
is such that 

A(H 2 ) HG = H 2 . 

Then for arbitrary Hi the problem SCSP can be solved in probabilistic polynomial time 
(in n and in log q) by the linear algebra technique, provided that n is less than q/2. Indeed, 
in this case the solution of the linear system hf — gh = with respect to h £ A(H 2 ) is an 
element of H 2 with a great probability. (From [3] it follows that in this case the problem 
SCSP can be solved efficiently even by a deterministic algorithm.) 

It seems that the problem SCSP remains difficult when G is restricted to subgroups 
of the group GL(V, R) of all invertible i?-linear transformations of the free -R-module V 
where R is a finite commutative ring. To see this we consider the Linear Transporter 
Problem on the intractability of which the second protocol is based. 

Linear Transporter Problem (LTP). Let R be a commutative ring, V be an R-module 
and G < GL(V,R). Given u £ V and v £ u G = {u 9 : g £ G} find g £ G such that 
v = u 9 . 

A special case of (LTP) is the Discrete Logarithm Problem. Indeed, take V = Z* with 
p being a prime. Then V can be considered as an one-dimensional module over the ring 
R = End(C) = Z p _! (with respect to taking the power v >— > v n where v £ V, n £ Z p _i). 
Choosing u to be a generator of the group V we come to the Discrete Logarithm Problem. 

Preserving the notation of LTP set T{V) = {T v : x 1— > x + v , v , x £ V} to be the 
translation group of the -R-module V. Then obviously 

v = u 9 & T v = g" l T u g, u, v £ V, g £ GL(C, R). 

So the problem LTP is the special case of the problem SCSP with G = AGL(V, R), 
Hi = T(V) and H 2 = GL(V,R). (Here AGL(C, R) = T(V)GL(V,R) is the group of 
all affine transformations of V.) This shows that SCSP is at least as hard as LTP. In 
particular, this construction gives us a family of groups for which the problem SCSP 
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turns to be at least as hard as the Discrete Logarithm Problem. A general technique to 
construct groups of this kind will be given in Section El 

2 Homomorphic cryptosystems over groups 

2.1 A general scheme. A homomorphic cryptosystem is a probabilistic public key 
scheme (in the sense of |7]) in which the spaces of plaintext messages and ciphertexts 
are groups H k and Gk respectively, depending on a security parameter k and such that 
its decryption function 

fk '■ Gk — > Hk (2) 

is an epimorphism for all k. Usually, in a homomorphic cryptosystem the public key 
includes generator sets Xk and Yk of the groups Gk and Hk, and some set Rk C Xk such 
that Yk C fk(Rk) — {fk(g) '■ 9 £ Rk}- Besides, it is assumed that there are publically 
known /c 0(1 )-algorithms to solve the following problems: 

(1) given two elements a, b of Gk (resp. Hk) find the element ab^ 1 , 

(2) given y G Yjt find an element of the set Rk PI f k {y), 

(3) generate a random element of the group ker(/^) 

where sizes of all elements are assumed to be at most k. Under these assumptions the en- 
cryption can be performed in time as follows. First, given a message h = y± ■ ■ ■ y m £ 
Hk with yi G Yk and m being a natural number at most k°^\ Bob computes in time 
polynomial in k an element r = r\- ■ ■ r m G Gk such that rj G Rk and fk{ r i) — Hi for all i. 
Second, Bob mixes r with random elements gi, ■ • • , g m +i £ Gk belonging to the kernel of 
the homomorphism fk and outputs the element g = g\T\g<i ■ ■ ■ g m Tmgm+i as the ciphertext 
of h. Alice being able to compute fk efficiently performs the decoding as follows: 

fk(g) = fk(gmg2 ■ ■ ■ gmr m g m+l ) = / fc (r x ) • • • f k (r m ) = yi---y m = h. 

The key point of such a system is to choose a presentation of the group Gk and the 
epimorphism fk in order to provide the inverse of fk to be a trapdoor function. The exact 
definition of homomorphic public-key cryptosystems and a survey of constructions can be 

found in manu. 

One way to implement the general concept of a homomorphic cryptosystem is to take 
Gk to be a subgroup of a certain group F such that the group operations in F can be 
performed in time polynomial in the size of operands. In the cryptosystems from JUj 
and [TT] the group F was taken as a free product of abelian groups and a modular group, 
respectively. In these cryptosystems the restriction of the mapping f k to the set Rk was 
known publically and one can produce efficiently random k ^-size elements of the group 
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ker(/fc). In fact, the security of these cryptosystems was based on the difficulty of the 
membership problem (see below) for special subgroups of the group Gk- In the next 
subsection we present a new homomorphic public-key cryptosystem of this kind (but with 
another trapdoor). 

2.2 A new homomorphic scheme. Let H = (Y; TV) be a finitely presented group 
generated by the set Y of cardinality k > 2 with 1Z C Wy as the set of relations. As the 
group F mentioned above we take the free group (Y). For a permutation a G Sym(Y) 
denote by <p a the automorphism of the group F induced by a. Set 

X = X a = W-\r y yr' y ): y G Y} (3) 

where r y and r' y are randomly chosen words of size 0(k) belonging to the set W-n C Wy. 
Then G = (X) is a subgroup of the group F. Moreover, the mapping f a :G^H defined 
by a commutative diagram 



G H 



id 



G 



P (4) 



where p : F —>■ H is the epimorphism induced by the mapping idy, is an epimorphism 
such that given x G X we have (see (J3J)): 

f a (x) = p((p a (x)) = p{^ a {^ l {r y yr' y ))) = p(r y yr' y ) = p{r y )p{y)p(r' y ) = p{y) = y 

where y is the element of Y for which x = (f>~ x {r y yr' y } (see (jSJ)). In particular, f a (X) = Y 
and the restriction of f a to X is a bijection. This enables us to construct a homomorphic 
cryptosystem as follows. 

Secret Key: the permutation a G Sym(Y). 

Public Key: a natural number k > 2, a group H = (Y;1Z) with |Y| = k, a subgroup 
G = (X a ) of the free group F = (Y), and the bijection / : X a — ► Y coinciding with the 
restriction of the homomorphism / CT to 

Encryption: a message M = y^ ■ ■ -y^ G H where y^ G Y ± , is encrypted by the element 

E(M) = /-^si^s'J ■ • • r\s t y it s' t ) e G 

where Sj and s- are random words of the set Wn C IYy of size 0(k), and for a word 
w = • • -y • • • E Wy we set / _1 (w) = ■ ■ • f~ l {y) 

Decryption: a ciphertext C = y^ ■ ■ ■ y it e G C F where y^. G Y 1 * 1 , is decrypted to 
D{C) = yl---yleH. 
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To prove the correctness of the decryption we note that / = (U)\x , U = p(<P(t)\g, 
and <f a (y) = y a for all y G Y (see (JH)). Since obviously = <p a -i, we have 

D(E(y h ■ ■ ■ y k )) = D{f' 1 (s l y il s 1 ) ■ ■ ■ f~ l (s t y it s' t )) = D^-i^y^ ■ ■ ■ s t y k s' t )) = 

fa^(s 1 y il s' 1 )' 7 ■ ■ •ip tT -i{s t y it s' t y = ■ ■ ■ s t y it s' t = y h ■ ■ -y it . 

Clearly, that both encryption and decryption algorithms are polynomial-time in the size 
of the input words. 

The security of the homomorphic cryptosystem will be discussed in the next subsection. 
Here we only make several remarks on the possible implementations. First, we note that 
it is not necessary to work with words; instead of this one can use a matrix representation 
of the group F (see JI]). Next, to choose the set Y so that \Y\ = k, one can take any 
set S of generators of H and add to it k — \S\ random elements of H whenever \S\ < k. 
Finally, as in Section any implementation of the above cryptosystem must be supported 
by sufficiently large class of candidates for groups H. We will return to this problem in 
Section El 

2.3 On the security of homomorphic schemes. Concerning the security of the ho- 
momorphic cryptosystem suppose first that the order of the group H is at most k°^ (e.g. 
such an assumption was done in JU|). Then using the generator set Y of H one can list 
all the elements hi, . . . , h m of this group in time k°^ and then to find within the same 
time a set {gi, . . . ,g m } of distinct representatives of right cosets of G by G a = ker(/ CT ) 
(one can set = f~ l {hi) for all i). Now if an adversary Charlie could recognize efficiently 
the elements of G belonging to G ai then he would efficiently compute f a (g) for all g G G 
due to the formulae 

U{g) = fa{9i) & ggi 1 G G a 

where i G {1, . . . , m}. Thus in this case the security of our cryptosystem is based on the 
intractability of the following problem: 

Membership Testing (MT). Given a group F and its subgroup G test whether a given 
g G F belongs to G. 

Suppose now that the order of H to be arbitrary. Then a quite natural way to break 
the cryptosystem is to find an expression of any g G G in the terms of generators belonging 
to the set X a (the attack of this kind was considered in [H]). Indeed, if Charlie could 
find efficiently for any element g G G an expression g = X\ - ■ ■ x m where Xj G for all i, 
then he would efficiently compute f a (g) due to formulae 

fM = Ufa) ■ ■ ■ U(x m ) = ffa) ■ ■ ■ f(x m ) 

(we recall that the bijection / : X a — > Y is given publically). Thus in this case we come 
to the presentation problem (see The MT problem and the presentation problem 
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are closely related each to other (but generally could be not polynomial-time equivalent) 
and one can combine them in the following well-known problem of computational group 
theory (see [3]). 

Constructive Membership Testing (CMT). Given a group F and its subgroup G 
generated by a set X find an expression of a given g G F as a word in X , or determine 
that g £ G. 

Last two decades a great attention was paid to CMT with different presentations of 
the group G. For example, if F is a subgroup of the symmetric group of degree n > 1, 
then the CMT can be solved in time n olyl > by the sift algorithm (see e.g. ^3]). In the case 
of groups F = GL(n, F) where F is an algebraic number field, there exists an effective 
Las Vegas algorithm solving CMT 3J. However, for n — 1 and F being a finite field, 
CMT is nothing else but the the Discrete Logarithm Problem. In |3] it was conjectured 
that CMT is difficult whenever the group G either involves a large abelian group as a 
quotient of a normal subgroup or has nonabelian composition factors which require large 
degree permutation representations. Finally, the problem becomes much more difficult if 
we take F = GL(n, R) the group of n x n invertible matrices over a ring R. In this case 
the problem is undecidable for n = 4 and R = Z (see [T5]). 




3 Cryptographical generation of groups 

3.1 A general scheme. We begin with a general scheme to construct a vast family of 
groups and homomorphisms supporting both key agreement protocols of Section Q and 
homomorphic cryptosystems of Section|21 Let Q be a class of groups closed with respect to 
a set of group-theoretical operations of different arities (like direct or wreath products). 
For an integer s > 1 we denote by S a set of all operations of arity s belonging to 0. 
For a set Q C Q we define recursively a class V(Qo, 0) of pairs (G,T) where G G Q and 
T is a rooted labeled tree, as follows: 

Base of recursion: any pair (G, T) with G G Go and T being the one-point tree with 
root labeled by G, belongs to V(Qo, 0). 

Recursive step: given pairs (Gi,Ti), . . . , (G S ,T S ) G V(Qo, 0) and an operation o G S , 
the class V(Qo, 0) contains the pair (G, T) where G = o(Gi, . . . , G s ) and T is the tree 
obtained from Ti, . . . , T s by adding a new root labeled by o and the sons being the roots 



Let (G, T) G V(Qo, 0)- Then obviously G G Q and the derivation tree T of G provides 
the constructive proof for this membership. The group G is uniquely determined by T 
and we call it the group associated with T. The fact, that a derivation tree is an ordinary 
rooted tree the leaves and the internal vertices of which are labeled by elements of Qq and 




of Ti 



T 

. . . , ± s . 
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respectively, enables us to choose a random derivation tree of a fixed size. 

Suppose from now on that all the groups of Q are given in a certain way (e.g., one can 
take as Q a class of matrix groups given by generator sets). We assume also that for each 
operation o G S and groups G\, . . . , G s G G, the size L(G) of the presentation of the 
group G = o(Gi, . . . , G s ) is at most O(L) where L = Yli=i L(Gi) and the group G can be 
constructed from G\, . . . , G s in time L olyl \ Let us define a size L(T) of a derivation tree 
T to be the sum of the sizes of all labels of T; thus L(T) includes the sizes of the groups 
assigned to the leaves of T together with the number of edges of T. Then the size of any 
pair (G,T) G V(Go, 0) is 0(L(T)), and the knowledge of T enables us to find G in time 
polynomial in L(T). 

One of the problems arising in constructions of group-theoretical public key cryptosys- 
tems is to find an efficient algorithm to produce a random group (or a collection of groups) 
belonging to a special class Q and with a given size L of the presentation. Such a group 
G must be equipped with a private key providing an efficient solution of a certain com- 
putational problem for G that is supposedly difficult in the class Q without knowledge of 
a private key. Our approach to the above problem is to choose an appropriate class Go 
of groups, a set of group-theoretical operations, and then to generate instances for the 
cryptosystem in question as follows: 

Step 1: given a security parameter L choose randomly groups Gi, . . . ,G t G Go, such that 
J2l 1 L(G l ) = 0(L); 

Step 2: choose randomly a rooted labeled tree T of size O(L) and with t leaves being 
labeled by G±, . . . , Gt, 

Step 3: compute the group G associated with T (i.e. (G,T) G V(Go, 0)); 

Step 4: output the group G as a public key and the labeled tree T as a secret key. 

Denote by G* the class of groups G such that (G, T) G V(Qo, 0) for some labeled tree 
T. Then the secrecy of the key T is based on the intractability of the following problem: 
given G G G* find a derivation tree T associated with G. A special case of this problem 
will be considered in Section EP1 

For a homomorphic cryptosystem the above scheme is not sufficient because together 
with the group G we have to provide a group H and a secret homomorphism / : G — > H. 
To this end suppose that each group G G Go is equipped with a set M(G) of pairs 
(H, f) where H G Go and / : G — > H is a homomorphism. We also assume that given 
homomorphisms fi G{ —> Hi with Gi,Hi G G* for i = 1, . . . , s, and an operation o G S 
there exists an efficiently computed homomorphism f : G —> H where G = o(G\, . . . , G s ) 
and H = o(H 1 , . . . ,H S ) such that f\d = f% for all i (here we suppose in addition that 
Gi is a subgroup of G). This homomorphism is denoted by o(/i, . . . , f s ). In this notation 
the set M.{Goi$) of instances (G,f) for a homomorphic cryptosystem can be defined 
recursively as follows: 
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Base of recursion: any pair (G, /) with G G Go and / G M(G) belongs to the set 
M(g o ,0); 

Recursion step: given pairs (Gi, f\), . . . , (G s , f s ) G M.(Go,0) and an operation 
o G S , the class A4(Go,0) contains the pair (G,f) where G = o(G\, . . . ,G S ) and 
/ = o(/i,...,/ a ). 

We observe, that in the process of constructing the homomorphism / : G —>■ H we 
also produce the derivation trees of the groups G and H. A realization of these general 
schemes in finite matrix groups will be considered in the next subsection. 

3.2 Generating matrix groups. Let us define the classes Go, Q of groups and the set 
of operations. First, we set 

Q = U n Ur {G : G is a subgroup of GL(n, R)} 

where n and R run over natural numbers and finite commutative rings respectively. Thus 
any G G Q is a group of n x n invertible matrices with entries belonging to R for some 
n G N and some finite commutative ring R. We recall that any such ring is a direct 
sum of local commutative rings and each of the latter can be described via appropriate 
Galois ring: the Galois ring GK(p m , r) of characteristic p m and rank r is Z p m [x]/ (/) where 
/ G Z p ™ [x] is a monic polynomial of degree r whose image in Z p [x] is irreducible (see |T7j). 

Proposition 3.1 [T71 05] Let R be a finite commutative local ring of characteristic p m 
and ¥ = GF(p r ) the residue field of R. Then 

(1) R x = T x (1 R + Rad(-R)) where T is a cyclic group isomorphic to F x 7 

(2) the subring Rq of R generated by T is a Galois ring GR(p m ,r), 

(3) R is a homomorphic image of the ring R$[Xi, . . . ,X t ] where t is the minimal size 
of a generator set of the radical of R.m 

Proposition 3.2 Let p be a prime andm,r be natural numbers. Then 

(1) there exists the unique up to isomorphism Galois ring GK(p m ,r) of characteristic 
p m and rank r, 

(2) each element r G GR,(p m , r) is uniquely represented in the form r = tip 1 where 
U G T U {0} for all i, 

(3) given a G Aut(F) the mapping r i— >■ ^iP 1 where a is the automorphism of the 
group T induced by a (see statement (1) of Proposition fOj) . is an automorphism 
ofGR(p m ,r).m 
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Due to statements (2), (3) of Proposition 13.11 and statements (2) of Proposition 13.21 a 
representation of the finite commutative ring R (resp., the group G) can be chosen to be 
polynomial in log(|i?|) (resp. in n and log(|i?|)). We also admit a hidden representation of 
R in which the decomposition in local summands is not presented explicitly, for example 
the ring of residues modulo an integer can be completely given by indicating this integer. 

We define a set Go C G to be a class of classical simple (including abelian) subgroups 
G of the groups GL(n, F) where n <E N and F is a finite field. Any such group G G Go 
is given by a set of generators so that the Membership Testing Problem can be solved in 
time polynomial in n and in the bit size of F. (Indeed, any nonabelian classical matrix 
group can be given together with a suitable matrix representation which can be used 
for testing membership; for an abelian group of a prime order p one can use, e.g. the 
two-dimensional representation 

Z+^GL(2,p), x^fj fj (5) 

which gives a trivial membership testing algorithm). In fact, it is not necessary that 
Go contains all classical groups; one can form Go from the group of special types, e.g. 
PSL(n, F) or something like that. Since the elements of Go are parametrized by the tuples 
of naturals, one can efficiently choose a random group G G Go with a given size L{G) of 
presentation. 

The choice of the set of operations was inspired by the Aschbacher theorem j2] on 
classifying maximal subgroups of classical groups. Let us describe the operations. 

Changing the underlying ring. Let R be a finite commutative ring and R' be an 
extension of R. Then the natural monomorphism 

V? : GL{n,R) -> GL(n, R') 

gives an unary operation in G taking G G G to f{G). This operation can be performed 
efficiently whenever e.g. the embedding R to R' is given explicitly and the number d = 
[{R') + : R + ] is small. Another example is the extension of Z m to Z m / where m is a divisor 
of m' . Conversely, any embedding of the ring R' into the ring Mat(<i, R) induces the 
natural monomorphism 

<p' : GL(n,#) -> GL(nd,R) 

taking a matrix of GL(n, R') to the block matrix of GL(nd, R) with d 2 blocks of size n. 
(Such a situation arises e.g. when R' is a field of the order q d and R is its subfield of 
the order q, or when R' is isomorphic to the direct sum of d copies of R.) This produces 
another unary operation in G taking G G G to ip'(G). In order not to blow up the 
representation one should assume that d is small. In both cases the isomorphism type of 
the group G (as an abstract group) does not change, but the operations change it as a 
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linear group. In fact, our constructions start with matrix groups over a finite field F. To 
pass to rings one can use standard extensions with R — F and R' = Mat(m, R), and also 
with R = Mat(n,p) and R' = Mat(m, Z p d) with a prime p. 

Direct products. Suppose that groups G\, . . . , G s G Q are such that Gi < GL(rij, R) 
where n, G PJ and R is a finite commutative ring. Then 

G — Gi® ■ ■ ■ ®G S < GL(n, R) 

where n = rii=i n «5 an d we obtain an s-ary operation in Q. A set of generators for the 
group G can be efficiently constructed from the generating sets for G\, . . . , G s by means 
of the Kronecker product of the corresponding matrices. When R is a field the group G 
is irreducible iff so are the groups Gi, . . . , G s . (A matrix group G is called irreducible if 
the underlying linear space contains no nontrivial G-invariant subspaces.) 

Similarly, if m = rii, Gi n G\ = {I m } and G\ normalizes Gi where i = 1, . . . , s and G\ 
is the group generated by Gj, j ^ i, then G\ x • • • x G s is a subgroup of GL(m, R) which 
gives one more m-ary operation. 

Wreath products. The wreath product G I T of a group G and a permutation group 
r < Sym(m) is defined to be the semidirect product of the m-fold direct product G m = 
G x ■ • • x G by the group T acting on G m via coordinatewise permutations. If G < 
GL(n, R), then the group G I T has two natural linear representations obtained from the 
natural monomorphisms 

G m -> GL(nm, R), G m -> GL(n m , R), 

the first of which is induced by the m-fold direct sum of the underlying i?-module, whereas 
the second one is induced by the m-fold tensor product of it. The images of the group GlT 
are called the imprimitive and the product actions of the wreath product, respectively. 
Thus we obtain two more efficiently computable m-ary operations in Q. In the case of 
R being a field the resulting groups are always irreducible whenever G is irreducible and 
T is transitive. For our purpose it is enough to set V to be the symmetric group. More 
elaborated way could be based on the fact that any transitive group is obtained from 
the action of a group on the set of right cosets by some subgroup by means of right 
multiplications. 

Conjugations. An obvious unary operation in Q consists in the conjugation of a 
group G G GL(n, R) by means of a randomly chosen matrix from GL(n, R). Such an 
operation enables us to hide the form of a generator set of the group G. 

Let be the set of the above operations and Q* C Q be the set of all groups G such that 
(G,T) G V(Qo,0) for some rooted labeled tree T (see Subsection 13. In the following 
statement we consider the specializations of the problems MT (see Subsection 12 .3|) and 
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LTP (see Subsection I1.3J1 for the class Q* . In both cases we suppose that the group 
G G Q* is given by a set of generators. If G < GL(n, R) for a certain n G N and for 
a finite commutative ring i?, then in the case of LTP we set V to be the standard free 
R- module of dimension n on which the group GL(n, R) acts, whereas for MT problem we 
set F = GL(n,R). 

Lemma 3.3 Let G G Q* . Then given a derivation tree of G the problems MT and LTP 
can be solved in time polynomial in L(G). 

Proof. Let T be a derivation tree of G. Then the labels of the leaves of T are the groups 
G\, . . . , Gt G §q. Due to the choice of Qq the problems MT and LTP can be solved for the 
group Gi in time polynomial in L(Gi) for i — 1, . . . , t. Since L(G) = L(T)°W, it suffices 
to verify that by means of the tree T the problems can be reduced in time L(T)°^ to 
the corresponding problems for G\, . . . , Gt- For this purpose let us consider, for instance, 
the reduction in the case of the primitive wreath product G = H \ T with H < GL(n, R) 
and T = Sym(m) (other operations from on groups are treated in a similar way). Then 
G < GL(n m , R) and since T is given, we know the decomposition 

V = U ® ■ ■ ■ ® U (m times) 

where V and U are the standard i?-modules for groups GL(n m , R) and GL(n, R) respec- 
tively. Any element g G G can be represented as the pair (h, k) G H m x Sym(m) such 
that 

(u 1 ,..., Um y = (u':;\...,ut m ) (6) 

where h = (hi, . . . , h m ) and ij = j k for j = 1, . . . , m. Now the permutation k can be 
efficiently computed from the elements of the form (0 R , . . . , 1 R , . . . , 0#) 9 (with 1 R being 
the unique nonzero component in a certain place). So the element h = gg^ 1 also can 
be found efficiently where is the element of GL(V) = GL(n m , R) corresponding to k 
(this element acts on V exactly by permuting coordinates according to k). In particular, 
this provides a polynomial time reduction of the MT problem for G to the corresponding 
problem for H . 

Next, proceeding to the LTP problem let v G u G for some u,v G V. Denote by D 
the bipartite graph with parts being the multisets {ui, . . . , u m } and {v i, . . . , v m } and the 
edges being the pairs (ui, Vj) for which Vi G (uj) H . Then from (jHJ) it follows that there is a 
one to one correspondence between the matchings {(ui,Vj { ) : % — 1, . . . , m} of the graph 
D and the set {k G T : v = u 9 with g = (h,k) G G for some h G H m }. Since the problem 
of finding a matching of a bipartite graph can be solved efficiently, we see that the LTP 
problem for G is polynomial time reducible to the corresponding problem for H.u 

A natural way to apply our construction to the key agreement protocol is to choose a 
random group G G Q* of a prescribed size and then choose random subgroups Ga and Gb 
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of G (see ((H)). These groups can be specified by sets of generators constructed as follows: 

Step 1. Let S be the set of leaves of the derivation tree of the group G. For each s G S 
take random subsets Xa(s) and Xb{s) of the group H s associated with s. 

Step 2. Using the natural embedding h — > g^ of H s into G output Xa = {g x : x G 
Xa(s), s G S} and Xb = {g x '■ x G Xb(s), s G S} as the generator sets of Ga and Gb 
respectively. 

Thus, the constructing of the groups Ga and Gb is performed simultaneously with the 
constructing the group G. (In fact, all we need is the embedding of each group assigned 
to a leaf of the derivation tree of the group G into G.) In this way it is possible to control 
some properties of the groups, for instance, to avoid the situation when Ga centralizes 
Gb (then the common key coincides with 1q and so is not secure). 

Applying our construction to design homomorphic cryptosystems is more delicate. 
First of all we define the set M(G) for each group G < GL(n, R) for some n G N and 
some finite commutative ring R (note that this covers the case G G Qq and also allows 
one to produce homomorphisms in one more way: replacing Q Q by a bigger subclass of 
Q). Namely, any automorphism o G Aut(i?) induces a homomorphism 

fa'-G > G 17 , A i > A u 

where the matrix A u G GL(n, R) is obtained from the matrix A G GL(n, R) by entry- wise 
applying of a. To choose a we observe that R = (Bi^iRi where each Ri is a finite local 
commutative ring. Any automorphism of the residue field of the ring Ri can be lifted to 
the automorphism of this ring (statement (3) of Proposition ^. 2|) . In the representation of 
the Galois ring as a quotient ring of a ring of polynomials this lifting can be done efficiently 
Taking any collection {<7j}j e / one can construct the automorphism a G Aut(i?) such that 
a\n. = Oi for all %. The set of such automorphisms we denote by Auto(-R) (in the case of 
R being a field this group coincides with Aut(i?)). Set 

M(G) = f U{f a : a G Auto(-R)} (7) 

where fo is a trivial homomorphism taking any element of G to the identity matrix of 
GL(n, R). Then assuming that the ring R is given explicitly, one can choose a random 
element of Ai(G) in time polynomial in L(G). 

To provide the recursive step in constructing homomorphisms take o G S , s > 1. 
Suppose first that s = 1. Then o is an unary operation, i.e. either it changes the 
underlying ring R of a group G < GL(n, R), or o is a conjugation. Given a homomorphism 
/ : G — » H with H < GL(n, R) we set o(f) to be the composition oof. Let now s > 1 
and fi : G j — > ii/j be a homomorphism with G^Hi G for z = 1, . . . , s. Then there exists 
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the natural canonical homomorphism 



o(f 1 ,...,f s ):o(G 1 ,...,G s )^o(H 1 ,...,H s ) 

coinciding with /j on the group Gi which in this case is a subgroup of the group 
o(Gi, . . . , G 8 ). In any case, the resulting homomorphism is efficiently computable (we 
recall that we represent a homomorphism by listing explicitly the images of the genera- 
tors). The above discussion shows that the following statement holds. 

Lemma 3.4 Let f : G — > H be a homomorphism constructed in the above way where 
G,H G G* ■ Then given the derivation tree of G one can find f(g) for g G G in time 
polynomial in L(G) and the size of g. ■ 

3.3 Secure generation. Let us fix the classes Go,G,G*, the set of operations and 
the sets M. (G) for G G Go as in Subsection 13.21 Then due to Lemmas 13.31 and 13.41 one 
can construct groups G G G* to realize both key agreement protocols and homomorphic 
cryptosystems in which the group G and the derivation tree T of it play the roles of public 
and secret keys, respectively. The security of such systems is based on the difficulty of 
the following problem. 

Decomposition Problem. Given a group G G G* find a derivation tree T of G. 

This problem arises in connection with a computational version of the above mentioned 
Aschbacher's theorem. A number of practical algorithms (without complexity bounds) 
for Decomposition Problem are known (see [Hj) but in general this problem seems to be 
difficult. Indeed, suppose that R = Z m where m = pq with p and q being two different 
primes. Denote by G p the cyclic matrix group of the order p in GL(2,p) (see (jSJ). 
Similarly, the group G q is defined. Then G p , G q G Go and 

G = G' p x G' q < GL(2, R) 

where G' p and G' q are the images of the groups G p and G q with respect to the natural 
embeddings GL(2,p) and GL(2, q) into GL(2, R). Thus the group G can be constructed in 
two steps: construct the groups G' p and G' q (the operation of changing the underlying ring), 
and set G = G' x x G' 2 (the operation of the direct product). This implies that G G G* ■ This 
shows that the integer factoring problem is a special case of the Decomposition Problem. 

Another strategy of Charlie could be to avoid solving the Decomposition Problem and 
to try solve the problems like LTP, SCSP or CMT directly. To prevent such an attack 
one can choose the leaves of the derivation tree of the group G to be the groups of the 
exponential size with respect to L(G). Then from the construction it follows that these 
groups will arise as the composition factors of G. However, for the groups with large 
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composition factors all the problems like LTP, SCSP or CMT seem to be difficult (see 
Subsections 11.31 and 12. 3|) . 

We mention one more attack of Charlie for the case of a homomorphic cryptosystem. 
Suppose we construct in the above way the homomorphism / : G — > H with G,H G Q*. 
We call the homomorphism linear Hit induces the ring homomorphism /' : A(G) — > A(H) 
where A(G) (resp. A(H)) is the subring of the underlying full matrix ring generated by 
G (resp. H). For a linear homomorphism the corresponding homomorphic cryptosystem 
can be easily broken whenever G < GL(n, R) where R = Z n for some n G N or R is a 
finite field (or, more generally, a direct sum of Galois rings). Indeed, in this case Charlie 
can find f(g) for g G G as follows. Take random generators gi, . . . ,g s of the group G 
and find a decomposition g = Yli=i c i9i with q G i? just involving linear algebra. Then 
/(#) = Si=i c if(.9i) due to the linearity of /. To prevent this attack one can take some 
initial homomorphisms at the leaves of the derivation tree to be elements of the group 
Auto(-R) (see (|7jl). Then the constructed homomorphism is not linear in general (e.g. if 
g G GL(n, F) with F being a field, and a G Aut(F), then generally (ag) a ^ ag a ). 

We complete the subsection by the following statement summarizing the above dis- 
cussion. 

Theorem 3.5 Assuming that the problems LTP, SCSP, CMT for matrix groups over 
finite commutative rings, as well as the Decomposition Problem are intractable, a secure 
two-party key agreement protocol and homomorphic cryptosystem can be implemented for 
these groups. ■ 

One of the consequences of this theorem is that by means of it one can construct 
encrypted simulation of a boolean circuit of the logarithmic depth (the details can be 
found in HD|). 

Final remarks 

One of the main problems in constructing homomorphic public-key cryptosystems con- 
sists in finding appropriate trapdoor functions. However, in the natural presentations of 
homomorphisms of algebraic structures the problem of breaking such a system is reduced 
to some variants of the CMT problem. On the other hand, there is the following result 
for matrix groups over finite fields. 

Theorem 3.6 O Theorem 6.1] Given K = (X) < GL(d,p e ) where X C GL(d,p e ), 
there is a Las Vegas algorithm that given any g G GL(d,p e ), decides whether g G K, and 
if g G K, then the algorithm produces a straight-line program with the input X, yielding 
g. The algorithm uses an oracle to compute discrete logarithms in fields of characteristic 
p with sizes up to p ed . In case when all of those composition factors of Lie type in 
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characteristic p are constructively recognizable with a Discrete Log oracle l , the running 
time is a polynomial in the input length \X\d 2 e logp, plus the time required for polynomially 
many calls to the Discrete Log oracle.m 

This theorem shows that having an oracle for the Discrete Logarithm, the membership 
problem can be solved in probabilistic polynomial time for matrix groups over finite fields. 
This means that at least for homomorphic public-key cryptosystems over such groups 
there is a little hope to find a trapdoor function different from functions the difficulty of 
inversion of which is based on the intractability of the Discrete Logarithm. However, only 
a little is known on the computational complexity of the membership problem for matrix 
groups over rings. So constructions over such groups seems to be more perspective from 
the point of view of algebraic (non-commutative) cryptography. 
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